Entry points for 3d trickplay

ABSTRACT

Providing entry points for 3D video data is described. An entry point unit ( 18 ) generates an entry point table by defining entry points in an incoming 3D video data stream and storing entry point addresses giving the location of the defined entry points. The video data stream comprises a multitude of sub-streams, which multitude encodes one stream of 3D video data and comprises at least one 2D sub-stream that independently encodes a 2D version of the 3D video data and at least one auxiliary sub-stream that dependently encodes part of the 3D video data. The entry points include main entry points in the 2D sub-stream and auxiliary entry points in the auxiliary sub-stream for enabling 3D trickplay of the 3D video data by retrieving and decoding nonadjacent fragments of the 2D sub-stream and retrieving and dependently decoding corresponding fragments of the auxiliary sub-stream.

FIELD OF THE INVENTION

The invention relates to a method of providing entry points for a videodata stream, the method comprising,

generating an entry point table;

defining entry points in the video data stream, in which the entrypoints are defined at a distance in time from each other;

storing the defined entry points in the entry point table by storingentry point addresses giving the location of the defined entry points.

The invention further relates to a device for providing entry points, adevice for reproducing video data, a signal, a method of rendering and acomputer program product.

The invention relates to the field of rendering 3D video data intrickplay mode, i.e. reproducing the 3D video with increased speed inforward or backward direction on a 3D display device.

BACKGROUND OF THE INVENTION

Devices for rendering 2D video data are known, for example video playerslike DVD players or set top boxes which provide digital video signals.The source device is to be coupled to a display device like a TV set ormonitor. Image data is transferred from the source device via a suitableinterface, preferably a high-speed digital interface like HDMI.Currently 3D enhanced devices for sourcing three dimensional (3D) imagedata are being proposed.

For 3D content, such as 3D movies or TV programs, additional controldata for enabling trickplay may be provided in combination with theimage data, for example a list of pointers to subsequent locations offrames that can be rendered at increased speed. Trickplay is anyrendering mode of the 3D video content at a speed different from theoriginal speed, such as fast forward or fast backward, or slow motion,in various speeds.

The document US 2006/0117357 describes a system for rendering 2D videodata in trickplay modes. A digital video signal is reproduced at varioustrick mode playback speeds. Frame indices associated with video framesof a digital video stream are monitored and a Group-of-Pictures (GOP)size is determined from the frame indices. One or more trick mode playspeed parameters are calculated based on the determined GOP size.Presentation of the video frames is controlled based on the calculatedtrick mode play speed parameters. In one embodiment, the trick mode playspeed parameters include a frame-skip count and a frame-repeat count.

For 3D content trickplay has to be developed also. One example of 3Dcontent is a two-dimensional image and an associated depth map. Anotherexample of 3D content is a plurality of two-dimensional images, e.g. thewell known stereoscopic content having a right eye image and a left eyeimage. Yet another example of 3D content is stereoscopic content havinga plurality of right eye images and left eye images, to be displayed ona multi-view display.

SUMMARY OF THE INVENTION

A problem of trickplay with 3D video is that the burden on the videodecoder increases as the decoder has to decode more frames in shortertime (for smooth trickplay). With stereoscopic video the decoder has todecode two or more streams and this increases burden as compared to 2D.In addition if the multiview video is encoded using dependent sub-streammultiview coding then decoding of the additional streams becomesdependent on the base view stream.

To provide 2D trickplay the Blu-ray Disc standard specifies an EntryPoint table (EP-map) for every elementary video stream. The video isencoded in frames of various types as defined in the well-known MPEGstandards. The table lists the location in the stream of points wheredecoding may start. Usually the entry points are at MPEG I frameboundaries. The table only lists the entry points for one stream, noaccount has been taken of the fact that several video streams may bedecoded simultaneously that are also dependent on each other.

It is an object of the invention to provide a system for 3D trickplay ina more convenient way.

For this purpose, according to a first aspect of the invention, in themethod as described in the opening paragraph, the video data streamcomprises a multitude of sub-streams, which multitude encodes one streamof 3D video data and comprises at least one 2D sub-stream thatindependently encodes a 2D version of the 3D video data and at least oneauxiliary sub-stream that dependently encodes part of the 3D video data;defining the entry points comprises defining main entry points in the 2Dsub-stream and auxiliary entry points in the auxiliary sub-stream forenabling 3D trickplay of the 3D video data by retrieving and decodingnon-adjacent fragments of the 2D sub-stream and retrieving anddependently decoding corresponding fragments of the auxiliarysub-stream.

For this purpose, according to a second aspect of the invention, thedevice for providing entry points for a video data stream comprisesmeans for generating an entry point table by defining entry points inthe video data stream, in which the entry points are defined at adistance in time from each other, and storing the defined entry pointsin the entry point table by storing entry point addresses giving thelocation of the defined entry points, wherein the video data streamcomprises a multitude of sub-streams, which multitude encodes one streamof 3D video data and comprises at least one 2D sub-stream thatindependently encodes a 2D version of the 3D video data and at least oneauxiliary sub-stream that dependently encodes part of the 3D video data,and the means for generating an entry point table are arranged fordefining main entry points in the 2D sub-stream and auxiliary entrypoints in the auxiliary sub-stream for enabling 3D trickplay of the 3Dvideo data by retrieving and decoding non-adjacent fragments of the 2Dsub-stream and retrieving and dependently decoding correspondingfragments of the auxiliary sub-stream.

For this purpose, according to a further aspect of the invention, thedevice for reproducing video data, comprises means for receiving a videodata stream and an entry point table as defined above, wherein the videodata stream comprises a multitude of sub-streams, which multitudeencodes one stream of 3D video data and comprises at least one 2Dsub-stream that independently encodes a 2D version of the 3D video dataand at least one auxiliary sub-stream that dependently encodes part ofthe 3D video data, and the entry points comprise main entry points inthe 2D sub-stream and auxiliary entry points in the auxiliarysub-stream; and the device comprises means for 3D trickplay of the 3Dvideo data by reproducing the 3D video data by, according to the entrypoint table, retrieving and decoding non-adjacent fragments of the 2Dsub-stream and retrieving and dependently decoding correspondingfragments of the auxiliary sub-stream.

For this purpose, according to a further aspect of the invention, thesignal that conveys video data comprises a video data stream havingentry points defined at a distance in time from each other, and an entrypoint table as defined above comprising the defined entry points bystored entry point addresses giving the location of the defined entrypoints, wherein the video data stream comprises a multitude ofsub-streams, which multitude encodes one stream of 3D video data andcomprises at least one 2D sub-stream that independently encodes a 2Dversion of the 3D video data and at least one auxiliary sub-stream thatdependently encodes part of the 3D video data; and the entry point tablecomprises main entry points in the 2D sub-stream and auxiliary entrypoints in the auxiliary sub-stream for enabling 3D trickplay of the 3Dvideo data by retrieving and decoding non-adjacent fragments of the 2Dsub-stream and retrieving and dependently decoding correspondingfragments of the auxiliary sub-stream.

For this purpose, according to a further aspect of the invention, themethod of rendering video data on the basis of the signal as definedabove comprises receiving a video data stream and an entry point tableas defined above, wherein the video data stream comprises a multitude ofsub-streams, which multitude encodes one stream of 3D video data andcomprises at least one 2D sub-stream that independently encodes a 2Dversion of the 3D video data and at least one auxiliary sub-stream thatdependently encodes part of the 3D video data, the entry points comprisemain entry points in the 2D sub-stream and auxiliary entry points in theauxiliary sub-stream; and the method comprises rendering 3D trickplay ofthe 3D video data by reproducing the 3D video data by, according to theentry point table, retrieving and decoding non-adjacent fragments of the2D sub-stream and retrieving and dependently decoding correspondingfragments of the auxiliary sub-stream.

The measures have the effect that trickplay of multiple substreamencoded 3D video data, e.g. multiview encoded video for Blu-ray Disc,now is provided with an extended entry point table. The traditionalentry point table provides a single entry point for a particular instantin a video stream. The entry point table according to the inventionprovides at least one further entry point for a particular instant thathas a main entry point for also directly accessing the correspondingauxiliary video stream. For example this is achieved by changing thedefinition of the entry point table such that the EP map related to thebase view video stream also contains the entry points for the associatedauxiliary streams, which by themselves cannot be decoded. When decodingof a particular fragment of 3D video to be reproduced in trickplay modethe necessary data of the main stream and the auxiliary stream can bedirectly accessed. Advantageously a viewer will not have to experiencedisturbing effects in the depth perception when not all sub-streams areproperly decoded or available due to missing references.

The invention is also based on the following recognition. The prior art2D trickplay system is not aware of the problems for 3D trickplay. Inparticular, for a single video stream a single set of entry points isprovided. However, in addition to a main substream that is independentlydecodable, one or more auxiliary sub-streams are present in a 3D videosignal. The inventors have seen that such sub-streams, which at normalreproduction speed, are only decodable in dependence of the main stream.Hence, traditionally, such auxiliary streams would not have entrypoints, because entry points in any non-decodable stream appear to bewithout any merit. Nevertheless the inventors have added entry point tothe non-decodable auxiliary stream. Only by providing both the main andauxiliary entry point addresses both streams can be conveniently decodedin non adjacent fragments for trickplay, because for such a fragment thecorresponding fragment of the auxiliary stream can immediately beretrieved according to the enhanced entry point table.

In an embodiment of the system the video data stream comprisesmulti-view 3D video data, which multi-view includes at least one leftview and one right view. Multiview 3D video provides multiple separateviews for the left and right eye. The multiple views of the 3D scenehave large overlap, and are usually dependently encoded, as explainedfor example in reference [1] or [2]. The enhanced entry point tableconveniently provides trickplay for such multiview 3D video streams.

In an embodiment of the system the video data stream comprises multipleauxiliary sub-streams and the entry points comprise auxiliary entrypoints only for a selected subset of said multiple auxiliary sub-streamsfor rendering a reduced version of the 3D video data during trickplay.Advantageously the size of the entry point table remains limited. Theembodiment is also based on the recognition that, during trickplay, somedegradation of the 3D video rendered is acceptable. For example, thenumber of views of multiview 3D video may be reduced by not decodingevery sub-stream, or transparency data may be ignored in a structured 3Dvideo format.

Further preferred embodiments of the method, 3D devices and signalaccording to the invention are given in the appended claims, disclosureof which is incorporated herein by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention will be apparent from andelucidated further with reference to the embodiments described by way ofexample in the following description and with reference to theaccompanying drawings, in which

FIG. 1 shows a 3-D video generation system,

FIG. 2 shows a multiview display,

FIG. 3 shows right and left eye view via lenticular lenses,

FIG. 4 shows a basic structure of a playlist,

FIG. 5 shows a system for displaying three dimensional (3D) video data,

FIG. 6 shows an entry point table indicator table,

FIG. 7 shows an enhanced entry point table indicator table,

FIG. 8 shows an enhanced stream type table,

FIG. 9 shows a 3D video stream having two sub-streams,

FIG. 10 shows a definition of an entry point map, and

FIG. 11 shows an entry point table for a combined main stream and asub-stream.

In the Figures, elements which correspond to elements already describedhave the same reference numerals.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 shows a 3-D video generation system. The 3-D video generationsystem comprises a pair of cameras, a right camera 11 and a left camera12, a 3D video processor 13 which generates a video signal 15 to bestored on a storage medium 14. The right camera and the left camera mayeach be a conventional camera. A depth scanner may be associated withthe left camera, comprising, for example, a laser beam that can besteered in various directions, and a sensor that detects reflections ofthe laser beam. Depth information may also be generated by computationfrom the camera information. The pair of cameras is directed towards ascene 10 as to capture a 3-D video of the scene. The scene 10 comprisesvarious objects, such as, for example, a person, a tree, a house, andthe sun in the sky. Each object has a given distance with respect to thepair of cameras, which may be regarded as a virtual observer watchingthe scene.

The 3D video processor may comprise, for example, aninstruction-executing device and a program memory into which a set ofinstructions has been loaded that define operations of the 3D videoprocessor, which will be described hereinafter. The storage medium 14may be in the form of, for example, a hard disk, a writable opticaldisk, a mastering system for manufacturing read-only type optical discsor a solid-state memory.

The 3-D video generation system basically operates as follows. The pairof cameras provides a basic 3-D video of the scene, which is formed by asequence of picture pairs. A picture pair comprises a right picture anda left picture. The right picture, which is captured by the rightcamera, is intended for the right eye of a human observer. The leftpicture, which is captured by the left camera, is intended for the lefteye of a human observer.

The right camera and the left camera have a particular positionalrelationship with respect to each other. This positional relationshipmay be defined by a typical rendering context in terms of, for example,screen size and a viewing distance. For example, the basic 3-D video,which comprises a sequence of right pictures and a sequence of leftpictures that are interrelated, may be intended for display in a cinemawith a typical screen size of 12 meters and a typical viewing distanceof 18 meters. A multiview 3D video data stream may be generated from thecamera and/or depth information. Multiview 3D video provides multipleseparate views for the left and right eye. The multiple views of the 3Dscene have large overlap, and are usually dependently encoded, asexplained for example in reference [1] of [2].

A different 3D format is based on two views using a 2D image and anadditional depth image, a so called depth map, which conveys informationabout the depth of objects in the 2D image. The format calledimage+depth is different in that it is a combination of a 2D image witha so called “depth”, or disparity map. This is a gray scale image,whereby the gray scale value of a pixel indicates the amount ofdisparity (or depth in case of a depth map) for the corresponding pixelin the associated 2D image. The display device uses the disparity, depthor parallax map to calculate the additional views taking the 2D image asinput. This may be done in a variety of ways, in the simplest form it isa matter of shifting pixels to the left or right dependent on thedisparity value associated to those pixels. Reference [3] gives anexcellent overview of the technology.

In the system shown in FIG. 1 the 3D video processor 13 has an entrypoint unit 18 for processing the incoming 3D video data and generate anentry point table for 3D trickplay mode. The entry point unit isarranged for defining entry points in the video data stream. The entrypoints are stored in the entry point table. The entry points are definedin the video data stream at a distance in time from each other.Subsequently the defined entry points are stored in the entry pointtable, e.g. by storing entry point addresses giving the location of thedefined entry points. In 3D video data stream formats the video datastream usually comprises a multitude of sub-streams, which multitudeencodes one stream of 3D video data and comprises at least one 2Dsub-stream that independently encodes a 2D version of the 3D video dataand at least one auxiliary sub-stream that dependently encodes part ofthe 3D video data. For example, the part may be a right view (dependingon an independently encoded left view stream), or a depth map. For sucha 3D video stream the entry points are generated to comprise main entrypoints in the 2D sub-stream and auxiliary entry points in the auxiliarysub-stream for enabling 3D trickplay of the 3D video data.

During rendering, selected fragments of the main (2D) sub-stream areretrieved based on the main entry points and decoded as non-adjacentfragments of the 2D sub-stream. Subsequently parts of the auxiliary,dependent sub-stream, which correspond to the selected parts of the 2Dsubstream, are retrieved based on the auxiliary entry points anddependently decoded as fragments of the auxiliary sub-stream.

FIG. 2 shows a multiview display 21, which uses lenticular lenses 22 infront of a LCD screen to generate a different view for the left andright eye. Interleaving two images shot from a slightly different anglecreates the 3D perception. This effect is based on binocular disparity,the left and right eye normally see an object from a slightly differentangle. These are fused together through accommodation and convergenceand this acts as a powerful depth cue for the brain.

FIG. 3 shows right and left eye view via lenticular lenses 30. The righteye 32 only sees the left part of the pixel 33 and the left eye 31 seesthe right part. The pixel parts are called sub-pixels 34. Fusion of theright- and left part of an image in the human viewer throughaccommodation and convergence creates a depth cue by presenting a singlestereoscopic image. Multiple left and right views can be created bysubdividing each pixel in multiple sub-pixels.

For example, in contrast to FIG. 3 where only two interleaved images areshown, a practical display may use, for example, 9 interleaved images,which gives a wider range of view and contour to the image, asschematically indicated in FIG. 2. To drive such a type of displayrequires either image-plus-depth based video which is processed togenerate multiple views, or multiview encoded video. To this end theBlu-ray Disc standard may be extended to include support for such 3Dvideo streams. A player may then drive not only autostereoscopicdisplays, but also other types of stereo 3D displays such as a displaythat alternates views and that uses shutterglasses to separate the viewsfor both eyes individually, or in the future may even includeholographics displays.

An alternative to the lenticular screen is the Barrier display, whichuses a parallax barrier behind the LCD and in front the backlight toseparate the light from pixels in the LCD. The barrier is such that froma set position in front of the screen, the left eye sees differentpixels then the right eye. The barrier may also be between the LCD andthe human viewer so that pixels in a row of the display alternately arevisible by the left and right eye.

From experiments with trickplay of 3D video it has been found that thequality of the “3D depth” impression deteriorates during trickplay. Apossible explanation is that stereoscopic video demands a larger andlonger effort to the human optical system (accommodation andconvergence) than normal 2D video, in order for the brain to fuse thetwo images received by the eyes into a “3D” mental image. When thenumber of frames shown per second increases considerably duringtrickplay, the human optical system appears to be unable to catch upcompletely with the higher frame rate.

Another problem of trickplay with stereo 3D video is that the burden onthe video decoder increases as the decoder has to decode more frames inshorter time (for smooth trickplay). With stereoscopic video the decoderhas to decode two or more streams and this increases the problem ascompared to 2D. In addition if the multiview video is encoded usingscalable multiview coding as defined by MPEG then decoding of theadditional streams becomes dependent on the base view stream, thereforethe way in which trickplay can be done in the player must change. Suchstreams, which cannot be decoded independently are called auxiliarysub-streams in this document. Such streams are to be dependently decodedbased on the corresponding main stream.

In the following explanation an example of an entry point table isdiscussed with reference to the Blu-ray Disc system. It is noted thatthe entry point table can be applied to any 3D video system which isbased on main and auxiliary video streams, and details of the Blu-raydisc system are not required for implementing the invention. The Blu-rayDisc standard specifies an Entry Point table (including an entry pointmap: EP-map) for every elementary video stream. The entry point tabledefines table which lists the location in the stream of points wheredecoding may start. Usually these are at MPEG I frame boundaries. Thistable only lists the entry points for one stream, no account has beentaken of the fact that several video streams may be decodedsimultaneously that are also dependent on each other.

It has been found that depth perception during trickplay is improvedwhen skipping frames to create a kind of slideshow effect. Henceseparate, non adjacent, fragments of the original 3D video stream aredisplayed in a sequence. Surprisingly the more frames that where skippedthe better the perceived depth became. This is somewhat in contrast withnormal 2D video where smooth trickplay—whereby the decoder decodes allthe frames faster—is perceived as being better. This can be explained bytaking into account the fact that it takes time for the optical systemto fuse the two images from the eyes into one stereoscopic image(through accommodation and convergence) and generate a “3D” mentalimage. In normal life this is not a problem as depth perception relieson many factors and binocular disparity (stereopsis) is only effectivefor objects that are close to the viewer. For fast moving objects motionparallax plays a bigger role than occlusion. In a 3D display thishowever is a problem as the 3D effect relies mainly on binoculardisparity so for fast moving objects the depth perception is diminished.

To solve the above problem for trickplay it is necessary to define theentry points for the sequence of selected fragments that are to bereproduced in the respective trickplay mode, as described above.

In an embodiment the Entry Point table of Blu-ray Disc is extended toaccommodate the additionally defined entry points. This table now listsentry points for the video and provides the link between thetime-positions in the video and the positions in the file on disc. Theextension is such that in addition to an entry for the 2D video stream,the table now also lists the entry points for the second, auxiliaryvideo stream, which is encoded using scalable video coding and isdependent on the primary video stream for decoding. This second entryestablishes an association from every entry in the first stream to thecorresponding entry point in the second stream. This last may contain anI or a P frame, where the P frame may in turn reference the I frame fromthe primary stream. This approach is taken as when using the directPresentation Time Stamps (PTS) values. Note that a separate EP map forthe second stream may not work on its own as the auxiliary stream isonly dependently decodable, e.g. may contain only P or B frames at thesame PTS times. As such the auxiliary stream is not a valid stream whendecoded on its own. For example for multiview encoded video in Blu-rayDisc the entry point table may be extended and the way the Blu-rayplayer uses the EP-map is adapted to retrieve both the main entry pointsand the auxiliary entry points. The specification is enhanced such thatthe EP map related to the base view video stream also contains the entrypoints for the associated auxiliary streams, which by themselves cannotbe decoded.

The references [1] and [2] describe the principles behind jointly codedvideo streams and the associated transport format. For example, beforeencoding, the 3D views are interleaved and then coded using hierarchicalB frames. Before transport the bitstream is split into a primary streamand an auxiliary stream. This is done for backwards compatibility suchthat a 2D decoder can decode and use the primary stream and ignore theauxiliary stream. In a modified decoder the primary and auxiliary streamare interleaved again and decoded. This creates a problem for trickplayon Blu-ray disc whereby the primary and auxiliary stream are storedseparately on disc. To solve this it is required that the EP map tableis extended such that the player knows which clips, i.e. part of thestreams, of the primary and auxiliary stream must be interleaved anddecoded for display of the section of the video that the player hasskipped to. By the enhanced entry point table as proposed, this problemis solved.

FIG. 4 shows a basic structure of a playlist. The example is based on BDand the role that the EP-map 41 (entry point table in the controlinformation CPI) takes in this structure. For a certain PTS value theEP-map provides a logical address, e.g. the corresponding source packetnumber in the clip AV stream file which is an MPEG encoded elementarystream. The structure is further described with reference to FIGS. 6 to11.

FIG. 5 shows a system for displaying three dimensional (3D) video data.A 3D source device 50, e.g. a disc player, is coupled to a 3D displaydevice 53 for transferring a 3D display signal 56. The 3D source devicehas an input unit 51 for receiving image information. For example theinput unit device may include an optical disc unit 58 for retrievingvarious types of image information from an optical record carrier 54like a DVD or BluRay disc. Alternatively, the input unit may include anetwork interface unit 59 for coupling to a network 55, for example theinternet or a broadcast network, such device usually being called aset-top box. Image data may be retrieved from a remote media server 57.The source device may also be a satellite receiver, or a media serverdirectly providing the display signals, i.e. any suitable device thatoutputs a 3D display signal to be directly coupled to a display unit.

The 3D display device 53 is for displaying 3D image data. The device hasan input interface unit for receiving the 3D display signal 56 includingthe 3D image data transferred from the source device 10. The device hasa 3D display for displaying the processed image data, for example a dualor lenticular LCD. The display device 53 may be any type of stereoscopicdisplay, also called 3D display, and has a display depth range indicatedby arrow 44.

The 3D source device 50 has an image processing unit 52 coupled to theinput unit 51 for processing the image information for generating a 3Ddisplay signal 56 to be transferred via an output interface unit 12 tothe display device. The processing unit 52 is arranged for generatingthe image data included in the 3D display signal 56 for display on thedisplay device 13. The source device is provided with user controlelements, for controlling display parameters of the image data, such ascontrast or color parameter. The user control elements as such are wellknown, and may include a remote control unit having various buttonsand/or cursor control functions to control the various functions of the3D source device, such as normal playback and recording functions, andfor selecting trickplay modes, e.g. via direct buttons, or via agraphical user interface and/or menus.

The source device 50 has a trickplay processing unit 48 for processingthe 3D video data in trickplay mode. The 3D video data is reproducedduring trickplay by, according to the entry point table, retrieving anddecoding non-adjacent fragments of the 2D sub-stream and retrieving anddependently decoding corresponding fragments of the auxiliarysub-stream. The 2D sub-stream is independently decoded for therespective fragment, and the 3D information is added based on thecorresponding fragment of the auxiliary stream as retrieved from thevideo data stream based on the auxiliary entry point.

FIG. 5 further shows the record carrier 54 as a carrier of the 3D imagedata. The record carrier is disc-shaped and has a track and a centralhole. The track, constituted by a series of physically detectable marks,is arranged in accordance with a spiral or concentric pattern of turnsconstituting substantially parallel tracks on an information layer. Therecord carrier may be optically readable, called an optical disc, e.g. aCD, DVD or BD (Blue-ray Disc). The information is represented on theinformation layer by the optically detectable marks along the track,e.g. pits and lands. The track structure also comprises positioninformation, e.g. headers and addresses, for indication the location ofunits of information, usually called information blocks. The recordcarrier 54 carries information representing digitally encoded 3D videodata, for example encoded according to the MPEG2 or MPEG4 encodingsystem, in a predefined recording format like the DVD or BD format.

Described below is the relevant part of the syntax of an EP-map tablebased on the Blu-ray Disc specification. We propose to extend this tablesuch that it may contain also the entries of the associated streamswhich are dependent for decoding on the main stream listed in the top ofthe EP-map table.

In practice this will mean that for every auxiliary stream that isjointly coded with another stream there is an EP_map in the same tableas the stream that it is dependent on for being decoded. The reverse,i.e. an additional table for the auxiliary stream, is also possible andis more efficient in case of backwards compatibility with 2D decoding.In this case there is an EP-map for the clips containing the auxiliarystreams. In this EP-map there are also the entry point locations for thepart of the base view stream of which the entry point in the auxiliarystream is dependent on for decoding. In case of playback of multiviewencoded video the player then only needs to load the EP-map of theauxiliary stream and then has the access points for the base view streamwhich must be decoded to be able to decode the frame at the access pointof the auxiliary stream.

In detail a new EP_map is proposed that contains a mapping of entrypoints to file location for a multiple stream encoded 3D video stream.The Blu-ray Disc specification currently defines only one type of EP_mapthis is indicated in a table in the specification as shown below.

FIG. 6 shows an entry point table indicator table. The table showsexisting EP map types. The indicator values for indicating the EP maptype may be defined in a standard describing a recording format, e.g.Blu-ray disc. It is proposed to add a new type for multiview coded (MVC)3D video in this table called the “EP_map_MVC or some similar naminglike EP_map_ST for stereoscopic 3D video. This EP_MVC_map_type may beindicated by the value 2.

FIG. 7 shows an enhanced entry point table indicator table. The tableshows existing EP map types and the proposed new type for MVC 3D videoin this table called the “EP_map_MVC. In an embodiment the respective EPmap type is included in the EP map data structure when generating the 3Dvideo data stream, and transferred to a playback device. The playbackdevice can now easily detect the new EP map table type, and adapt thetrickplay operation to the respective EP map.

FIG. 8 shows an enhanced stream type table. Alternatively to FIGS. 6, 7the new EP-map is now indicated using the EP_stream_type value as shownin the table with a new value (8 in the table) for the type of streamreferenced in the EP_map. In an embodiment the respective EP stream typeis included in the sD video stream data structure when generating the 3Dvideo data stream, and transferred to a playback device. The playbackdevice can now easily detect the new EP stream type and retrieve theenhanced entry point table from the stream, and adapt the trickplayoperation to the enhanced entry point table.

FIG. 9 shows a 3D video stream having two sub-streams. The figure showsan example of MVC encoding of a section of two streams usinghierarchical B-pictures. The upper sequence marked L is an independentlydecodable 2D sub-stream, while the lower sequence marked R isdependently decodable, because it requires data from the first stream.An arrow indicates that data from the first I picture is used forencoding the first P picture of the lower substream.

In the example as shown in FIG. 9 there are three entry points in boththe L and R stream. In the L stream there is an I, B(T2) and B(T4)picture and in the R stream there is a P, B(T2) and B(T4) picture. TheB-pictures in between are non reference frames and cannot serve asentry-point. It is noted that, in practice, the distance between entrypoints will be substantially larger.

We will now continue by investigating what happens if the user wants tojump to the location T2. If decoding starts in T2 then the decoder mustalso have access to the I picture in T0 for the L stream and for the Rstream it must have access to the I-picture from the L stream and theP-picture from the R stream. So it requires the location of theI-picture in the L stream and the location of the P-picture in the Rstream. So it requires a temporal vector to the location of theP-picture and a spatial vector to the I-picture of the L frame.

On the disc the L and R stream each may be interleaved in differentsections on the disc or may be contained in one stream. Therefore both alocation in the file and a location on the disc may be needed for oneentry-point, as for one entry point information from both the L and Rstream is required as explained above. Hence a main entry point in thesub-stream L and an auxiliary entry point in the dependently decodablesubstream R are to be provided.

Therefore in detail we propose to extend the EP map for MVC encodedvideo such that each entry point contains two addresses, also calledvectors. One temporal vector points to the PTS and one spatial vectorpoints to a packet number of frames that serve as reference frame to theentry point.

FIG. 10 shows a definition of an entry point map, also called EP_mapsyntax. The table shows an example of the current EP-map extended foruse with MVC encoded video. The map comprises sub-tables for respectivesub-streams. It is noted that the table defines the data structure ofthe entry point table which is included with the 3D video data stream,e.g. in the control information on a record carrier such controlinformation CPI 41 in the Blu-ray disc format.

FIG. 11 shows an entry point table for a combined main stream and asub-stream, also called an EP map for one stream PID. In this embodimentno specific MVC section is added as shown in FIG. 10, but the table isextended with additional entries such that each entry point alsoindicates the list of packet numbers and PTS values in the dependentdata streams.

In an embodiment of a playback system for multi-view encoded 3D videotrickplay is arranged as follows. The 3D video stream has multipleauxiliary sub-streams and the entry points comprise auxiliary entrypoints only for a selected subset of said multiple auxiliarysub-streams. During trickplay a reduced version of the 3D video data isrendered by only decoding the sub-streams having the entry points.Advantageously the size of the entry point table remains limited.

Alternatively the decoder automatically reduces the number of views whenperforming trickplay to reduce the burden on the decoder. The number ofviews can be reduced dynamically in steps for increasing speeds, e.g.9-7-5-3-2. The respective entry points for the reduced number of viewsmay be retrieved from an entry point table. Alternatively a reducednumber of views may be generated during trickplay in a processing unitwhich produces said full multitude of views during standard speedplayback.

It is to be noted that the invention may be implemented in hardwareand/or software, using programmable components. A method forimplementing the invention has the processing steps corresponding to theprocessing of 3D video data elucidated with reference to FIG. 1.Although the invention has been mainly explained by embodiments usingoptical record carriers or the internet, the invention is also suitablefor any image interfacing environment, like a 3D personal computer [PC]display interface, or 3D media center PC coupled to a wireless 3Ddisplay device.

It is noted, that in this document the word ‘comprising’ does notexclude the presence of other elements or steps than those listed andthe word ‘a’ or ‘an’ preceding an element does not exclude the presenceof a plurality of such elements, that any reference signs do not limitthe scope of the claims, that the invention may be implemented by meansof both hardware and software, and that several ‘means’ or ‘units’ maybe represented by the same item of hardware or software, and a processormay fulfill the function of one or more units, possibly in cooperationwith hardware elements. Further, the invention is not limited to theembodiments, and lies in each and every novel feature or combination offeatures described above.

Reference [1]: “A novel Milti-View Video Coding Scheme Based on H.264;by GuopingLi, Yun He; ICICS-PCM 2003, 15-18 Dec. 2003, Singapore, IEEE0-7893-8185-8/03/$17.00”

Reference [2]: “Efficient Prediction Structures for Multi-View VideoCoding; by Philipp Merkle et al; IEEE 2007”

Reference [3]: “Depth image based rendering, compression andtransmission for a new approach on 3D TV” by Christoph Fehn (seehttp://iphome.hhi.de/fehn/Publications/fehn_EI2004.pdf)

1. Method of providing entry points for a video data stream, the methodcomprising, generating an entry point table; defining entry points inthe video data stream, in which the entry points are defined at adistance in time from each other; storing the defined entry points inthe entry point table by storing entry point addresses giving thelocation of the defined entry points, wherein the video data streamcomprises a multitude of sub-streams, which multitude encodes one streamof 3D video data and comprises at least one 2D sub-stream thatindependently encodes a 2D version of the 3D video data and at least oneauxiliary sub-stream that dependently encodes part of the 3D video data;defining the entry points comprises defining main entry points in the 2Dsub-stream and auxiliary entry points in the auxiliary sub-stream forenabling 3D trickplay of the 3D video data by retrieving and decodingnon-adjacent fragments of the 2D sub-stream and retrieving anddependently decoding corresponding fragments of the auxiliarysub-stream.
 2. Method as claimed in claim 1, wherein the video datastream comprises multi-view 3D video data, which multi-view includes atleast one left view and one right view.
 3. Method as claimed in claim 1,wherein the at least one auxiliary sub-stream comprises at least one ofa depth information data stream; a transparency information data stream;an occlusion information data stream.
 4. Method as claimed in claim 2,wherein the video data stream comprises multiple auxiliary sub-streamsand the entry points comprise auxiliary entry points only for a selectedsubset of said multiple auxiliary sub-streams' for rendering a reducedversion of the 3D video data during trickplay.
 5. Method as claimed inclaim 1, wherein the method comprises the steps of: generating a firstentry point sub-table, associated with the 2D sub-stream, and generatinga second entry point sub-table, associated with the auxiliarysub-stream, and forming the entry point table associated with the 3Dvideo data by including the sub-tables formed by the first entry pointsub-table and the second entry point sub-table.
 6. Method as claimed inclaim 1, wherein the method comprises the step of: defining, for eachentry point, a set of multiple entry point addresses, which include atleast a first entry point address to a main entry point and at least asecond entry point address to a corresponding auxiliary entry pointaddress.
 7. Method as claimed in claim 1, wherein the method comprisesproviding a record carrier having the data stream and the entry pointtable stored thereon.
 8. Device for providing entry points for a videodata stream, the apparatus comprising: means (18) for generating anentry point table by defining entry points in the video data stream, inwhich the entry points are defined at a distance in time from eachother, and storing the defined entry points in the entry point table bystoring entry point addresses giving the location of the defined entrypoints, wherein the video data stream comprises a multitude ofsub-streams, which multitude encodes one stream of 3D video data andcomprises at least one 2D sub-stream that independently encodes a 2Dversion of the 3D video data and at least one auxiliary sub-stream thatdependently encodes part of the 3D video data, and the means (18) forgenerating an entry point table are arranged for defining main entrypoints in the 2D sub-stream and auxiliary entry points in the auxiliarysub-stream for enabling 3D trickplay of the 3D video data by retrievingand decoding non-adjacent fragments of the 2D sub-stream and retrievingand dependently decoding corresponding fragments of the auxiliarysub-stream.
 9. Device claimed in claim 8, wherein the device compriseswriting means for storing the video data stream and the entry pointtable on a record carrier.
 10. Device for reproducing video data, theapparatus comprising means (58, 59) for receiving a video data streamand an entry point table as defined in claim 1, wherein the video datastream comprises a multitude of sub-streams, which multitude encodes onestream of 3D video data and comprises at least one 2D sub-stream thatindependently encodes a 2D version of the 3D video data and at least oneauxiliary sub-stream that dependently encodes part of the 3D video data,and the entry points comprise main entry points in the 2D sub-stream andauxiliary entry points in the auxiliary sub-stream; and the devicecomprises means (48) for 3D trickplay of the 3D video data byreproducing the 3D video data by, according to the entry point table,retrieving and decoding non-adjacent fragments of the 2D sub-stream andretrieving and dependently decoding corresponding fragments of theauxiliary sub-stream.
 11. Device claimed in claim 10, wherein the devicecomprises reading means (58) for reading the video data stream and theentry point table from a record carrier.
 12. Signal (15) that conveysvideo data, the signal comprising: a video data stream having entrypoints defined at a distance in time from each other; an entry pointtable as defined in claim 1 comprising the defined entry points bystored entry point addresses giving the location of the defined entrypoints, wherein the video data stream comprises a multitude ofsub-streams, which multitude encodes one stream of 3D video data andcomprises at least one 2D sub-stream that independently encodes a 2Dversion of the 3D video data and at least one auxiliary sub-stream thatdependently encodes part of the 3D video data; and the entry point tablecomprises main entry points in the 2D sub-stream and auxiliary entrypoints in the auxiliary sub-stream for enabling 3D trickplay of the 3Dvideo data by retrieving and decoding non-adjacent fragments of the 2Dsub-stream and retrieving and dependently decoding correspondingfragments of the auxiliary sub-stream.
 13. Storage medium (14, 54)comprising a signal according to claim
 12. 14. Storage medium accordingto claim 13, wherein the storage medium is a record carrier (54) of anoptically readable type having a track having optically detectablemarks, optically detectable properties of the marks representing thesignal.
 15. Method of rendering video data on the basis of a signalaccording to claim 12, wherein the video data stream comprises amultitude of sub-streams, which multitude encodes one stream of 3D videodata and comprises at least one 2D sub-stream that independently encodesa 2D version of the 3D video data and at least one auxiliary sub-streamthat dependently encodes part of the 3D video data, the entry pointscomprise main entry points in the 2D sub-stream and auxiliary entrypoints in the auxiliary sub-stream; and the method comprises rendering3D trickplay of the 3D video data by reproducing the 3D video data by,according to the entry point table, retrieving and decoding non-adjacentfragments of the 2D sub-stream and retrieving and dependently decodingcorresponding fragments of the auxiliary sub-stream.
 16. Computerprogram product that comprises a set of instructions, which when loadedinto a processor, causes the processor to carry out the method asclaimed in claim 15.